Search AI Products and News

In today's rapid development of Large Language Models (LLM), the training and inference costs of models have increasingly become a focus of research and application. Recently, the Tencent Hunyuan team released an important study that delves into the 'Scaling Laws' of low-bit floating point quantization training, which refers to the principles governing the scale of floating point quantization training. The core of this research lies in exploring how to significantly reduce computational and storage costs without sacrificing performance by lowering the precision of the model.

OpenAI and DeepMind have differing views and approaches in the research of Scaling Laws. Scaling Laws can predict changes in loss when varying model parameters, data amounts, and computation resources. This competition will drive the development of artificial intelligence and shape the future of human-machine coexistence. Pre-training large language models involves trade-offs between model size, data quantity, and training costs. Scaling Laws can assist in optimizing design decisions. DeepMind argues that model size and data quantity should scale equally.